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[57] 



ABSTRACT 



According to this invention, an image processing apparatus 
which allows detailed observation of the point of interest 
upon zooming of a camera, includes an input unit (1000, 
1002) for inputting image information, and an encoding unit 
(1003, 1004) for encoding the input image information. 

34 Claims, 6 Drawing Sheets 
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IMAGE PROCESSING METHOD AND compeasatioD circuit 13 through a switch 12, and stores fee 

APPARATUS IN WHICH THE LEVEL OF sum data in an internal memory (not shown) of the motion 

ENCODING IS BASED ON THE LEVEL OF compensation circuit 13. 

OPTICAL ZOOM The image data S4 quantized by the quantization circuit 5 

BACKGROUND OF THE INVENTION 5 " sub J ccted t0 variable length coding by a variable length 

rp. B t . 1# , coding circuit (VLC) 6, and the codeddata is suppliedtoa 

The present invention relates to an image processing buffer 7. After the data is temporarily stored in the buffer 7, 

method and apparatus and, more particularly, to a method the data is supplied to a data multiplexing circuit 8, and is 

which comprises a communication terminal apparatus and multiplexed with audio data and other data while being 

an image input camera, and generates an image signal for 10 subjected to output control. The multiplexed data is supplied 

transmitting image information to another communication t o a line control unit 

terminal apparatus using a high-precision information sex- *m. * . „ , 

vice netwSk (eg., an ISDN) « k spccial-puiposc line ^f m 15 m ^« es * e 

_ . * . ^ pui^o^ uiic. state, and for example, coarsens/dtaunishes the quantization 

Conventionally, a method of communicating video step of the quantization circuit 5 in correspondence with the 
signals, audio signals, data signals, and the like between a 13 amount of data stored in the buffer 7 

plurality of remote points using ^-precision information when an input image includes, e.g.', a scene change, and 

service network such as an ISDN or a special-purpose line t u» ^/TtJ^T^ Ja it " " f' " 

is known i~~« y^y^ uuc the amount of the change exceeds the threshold value till 

' which motion compensation is possible, or when motion 

In this case, it is important to reduce the transmission rate compensation is unnecessary, the switch 3 and a switch 12 

of image data to be transmitted by removing redundant 20 are controlled to be switched to the input image side Then 

components of a video signal using an apparatus for encod- control is made in correspondence with an externally input 

ing and compressing a motion image. As a method of selection setting signal for selecting one of the image quality 

encoding and compressing it, a circuit with the arrangement and transmitting time priority modes upon transmission of 

shown in, e.g., FIG. 1 is popularly used. More specifically, motion images onto a transmission path 

the circuit shown in FIG. 1 tfvides each frame of an image * FIG. 2 is a block diagram showing an example of an 

in units of predetermined Mocks and orthogonally ^ proccssing a ^ atus whi ^ uses Z above- 

bansforms, quanUzes and encodes the Mocks to attain men tioned motion imTge encoding circuit, and includes a 

rnouon^ompensated mterframe prediction. camera controller used in, e.g^elevision meeting system. 

Referring to FIG. 1, a motion image video signal SI input a video telephone set or the like. More specificaUy, a circuit 

from an input terminal 1 has been converted into a digital 30 block for transferring image data input from a camera or 

signal in an arbitrary format, and is supplied to a motion audio data input from a microphone to a remote station or 

compensation circuit 13. The motion compensation circuit receiving and processing data from a remote station, and a 

13 compares the input image with the previous frame image block, called a codec, for performing other data processing 

stored in its internal memory. arc combined as a system. 

The motion compensation circuit 13 divides one frame 35 Referring to FIG. 2, reference numeral 201 denotes a 

image into a plurality of blocks each having an arbitrary CCD camera; 202, a camera input image signal processor, 

size, and detects their motion vectors. Upon detection of the 203, a camera controller; 204, a pan pod for determining the 

motion vectors, motion-compensated mterframe prediction camera direction; 205, an audio input microphone; 206, an 

is performed on the basis of the image data of the previous input audio signal processor; 207, a system controller; and 

frame, and predicted image data is output from the motion 608, a motion image encoding circuit 

compensation circuit 13. Reference numeral 209 denotes a line control unit for 

The output predicted image data is filtered through a receiving a camera control signal from a remote station, and 

two-dimensional spatial filter in a filter 14, and the filtered transmitting a status information signal. Although not shown 

signal is supplied to a subtracter 2. The subtracter 2 performs 45 in FIG. 2, the apparatus also includes a circuit for demodu- 

a subtraction between the motion image video signal SI lating and displaying image and audio signals from a remote 

input from the input terminal 1 and the predicted image data station. 

^ supplied from the filter 14. and outputs the resulting operation of the image processing apparatus with the 

difference signal to a switch 3. arrangement shown in FIG. 2 will be <Sed below. An 

The switch 3 supplies the output from the subtracter 2 as ^ image signal input by the CCD camera 201 is converted into 

predicted error data S3 to an orthogonal transform circuit, an outputtable video signal and is then converted into a 

eg., a discrete cosine transform (OCT) circuit 4. The pre- digital video signal in the image signal processor 202. The 

dieted error data S3 is orthogonally transformed by the DCT digital video signal is then supplied to the motion image 

circuit 4 in units of arbitrary blocks, and the transformed encoding circuit 608. The digital video signal is compressed 

data is supplied to a quantization circuit (Q) 5. 55 by the motion image encoding circuit 608, and the com- 

The quantization step of the quantization circuit 5 is pressed signal is supplied to the line control unit 209. 
determined by a system controller 15. The input predicted An audio signal is input from the microphone 205. The 
error data S3 is quantized on the basis of the quantization audio signal is amplified into an outputtable signal and is 
step determined by the system controller 15. compressed by digital conversion in the audio signal pro- 
Image data S4 quantized by the quantization circuit 5 is 60 cesser 206. The digital audio signal is then supplied to the 
supplied to an inverse quantization circuit 9, and is inversely motion image encoding circuit 608. Hie audio signal is 
quantized. Thereafter, the inversely quantized data is sup- multiplexed with image data and data from the system 
plied to an inverse orthogonal transform circuit 10, and is controller 207 in the motion image encoding circuit 608. 
subjected to inverse orthogonal transformation. The trans- The image signal, audio signal and various data input to 
formed data is then supplied to an adder 11. 63 the line control unit 209 via the above-mentioned circuits are 
The adder 11 adds reproduced and predicted error data S5 converted into transmission data to be sent to another 
to the predicted image data S2 supplied from the motion station, and the transmission data is transmitted via a line. 
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The camera 201 and the microphone 205 are controlled by input image urformauon; areadout meaas for -^8°«! ^ 

a control signal (not shown) inVut via the line control unit stored image information on the basis of jm 

2W The S signal has a^etermined format readout range control amount; and an encoding means for 

What control sSnal is input from an external apparatus encoding the readout image infonnauon. 

to ™ 1m Sol ffioVK supplied to and mulcted 3 Omer features and advantages of the ^< 

u> ujc iiuc wu« • will be apparent from the following description taken in 

by the system controller 207. c^iuwtioowith me accompanying drawings, in which like 

The system controller 207 supplies the interpreted contro S^^^^S* the samTor similar parts 

signal to the camera controller 203.When *° JL^lBS^ES 

is a signal associated with, eg., the pan and tilt operations iq uuvu * u 

of the camera, the camera controller 203 supplies the control BRIEF DESCRIPTION OF THE DRAWINGS 
signal to the pan pod 204 to change the direction of the 

camera 201 The accompanying drawings, which are incorporated in 

On the other hand, when the control signal is associated and constitute a part of the specification illustrate embodi- 

wim the zooming operation ox iris adjustment, the camera 15 ment of the invention and, together with *e description, 

controller 203 supplies the control signal to the camera 201 serve to explain the principles of the invention, 

to control it. Furthermore, when the control signal is asso- FIG. 1 is a block diagram showing the arrangement of the 

dated with the volume adjustment of the microphone 205, CD tire conventional image processing apparatus; 

the system controller 207 supplies the control signal to the pjQ 2 is a schematic block diagram showing the arrange- 
audio signal processor 206 to adjust the level of the output ^ ment of a motion image encoding circuit used in the con- 

signal ventional image processing apparatus; 

In the above-mentioned image processing apparatus, the pj G 3 ^ a schematic functional block diagram showing 

motion image encoding circuit 608 performs an operation tne arrangement 0 f an image processing apparatus according 
independently from the entire system. More specifically, it is 

to an embodiment of the present invention; 
possible to some extent that the quantization step size of the 25 nG 4 fe a block diagram showing the arrangement of the 

quantization circuit 5 in the motion image encoding circuit ' ^ ocessing apparatus according to the embodi- 

608 is changed, or the transmitting time or image quail* ^ £ H& 3; 

priority mode is selected by *^**^™^J* FIG. 5 is a schematic block diagram showing the arrange- 

an image upon execution of the output control of the buffer ^ ^ ^ ^ - mthcinmc 

7. 30 nrocessinc apparatus according to the embodiment shown in 

However, an externally input control signal can hardly SW™ 

attain flexible control corresponding to the current abjatton • ffie op^o,, me image 

or specific control inherent to the system For example B ^^lL am according to the embodiment shown in 

when an image signal from the camera 201 is received by a P°^\ s ^ «PP*"» S accorumg 
remote monitoring system, if zooming to a point of interest 35 mu. a 

is performed upon a change in an image, a small motion of DETAILED DESCRIPTION OF THE 

an image is converted into a large one. PREFERRED EMBODIMENT 

.22* SZStJlEZi STbSES THe arrangement of an image processing apparatus 
^frartKrib o to station in this state. <° according to an embodiment of the present invention wOl be 

cult to ascertain details 01 me snuauoo described below with reference to the accompanying draw- 

SUMMARY OF THE INVENTION tags. 

The present Invention has been made in consideration of FIG. 3 is a schematic functional block diagram showing 

processing method comprising: an input step of toputtmg l«*an ^"J^ 1 ^ compression ratio control 

image information on the basis of a predetermined zoom ett ?^^/^J^SToM 

omtool amount- and an encoding step of encoding the input unit 1004, and a line control unit im . _ 

SX^ofoa *e basis oYuIzocm contro! amount » The image input unit 1000 tepuU image J£ 

£^^*£^*£Z*!* s3^avit^ra n 1mfge , &Sg 1 S 

image processus method coming, an J"** « £Xed«g and A/D converting the input image. The input 

^ed imaee infonnation on me basis of a predetermined memory 1006 of the image input unit 1000. 

S^LlZol amount; and an encoding step of The input coatrol unit 1002 controls the ""PUtopcraUon of 

^SSX ieSEta5 information. image infonnation by the imageinput unit 1000, ix^con- 

T^CLToo^ * *o -Ul«d by providing a, trols the input operation in accc,ruance with an externally 

im Z r^ssine apparatus comprising: input means for «, input control signal Sc. 

Z^S^oSonm^ofa^ede^ed The ^^^^u^^s^ 

rotn^onttol amount; and an encoding means fox encoding signal (SI) input by the image input umt IfMT by a prede- 

^u^image^Zmation on thebasis of the zoom control termined encoding method. For example a md* ^g* 

me inpui in»»c encoding method for assigning a shorter length of a code to 

matting image ^formation; a storage means for storing the probability of occurrence is used. 
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J?L^^ 8 C0mp ^ sl0 'lf s 0 unit 1004 1015 in the motioo image encoding circuit 1208 inputs zoom 

Ir^Min^t iftftA*^^ operation of the image information encod- focal length data from the 3 system controller 1207 

ing unit 1003 in coirespondence with the control state of the through a zoom data line 1252 (step S2), and stores the input 

image pickup opera** of the image input unit 1000 by the data in its internal memory (not shown) (step S3). The flow 

input control unit 1002, thereby changing the compression 5 advances to step S4, knd the input focal* lt^ te* Z 

ratio upon encoding of input image information SI. compared with a predetermined wide-end focal length. 

The line control unit 1005 converts an image signal, audio Upon comparison of the data, it is checked first if the data 

signal and various data input via the above-mentioned is smaller man a r^terrnined focal length a (mm). If it is 

circuit into data to be transmitted to a remote point, and determined that the data is smaller than the focal length a 

outputs the converted data via a line. to (mm), Le., if YES in step S4, it is determined Aat the^ocal 

In this embodiment, the input control unit 1002 performs length data is closer to the wide-end focal length, and the 

zooming control of the video camera according to the flow advances to step S5 to set the quantization step in the 

control signal Sc, which designates an amount of the zoom- motion priority mode. 

m ^ 15 The quantization step is set as follows. The system 

In a zoom-in (enlargement) operation of the zooming controller 1015 in FIG. 5 supplies a table write command to 

control, the compression ratio upon encoding of image a quantization coefficient table memory 1016 to read out a 

information in the image information encoding unit 1004 is first quantization coefficient QF1 from the quantization 

decreased, and in a zoom-out (reduction) operation, the coefficient table memory 1016, and supplies the readout 
compression ratio upon encoding is increased. ^ coefficient to a quantization circuit 1005. Thus, the quanti- 

With this control, since the encoding operation of the zaa0D circuit 1005 sets a coarse quantization step in corre- 

image information encoding unit 1004 can be controlled in spondence with the first quantization coefficient QF1. 

correspondence with the input operation of image The flow advances to step S6, and the system controller 

Mormation, the image compression ratio can be changed in 1015 increase the frame rate by setting the data output rate 
synchronism with the zooming operation of the camera. In 25 of a buffer circuit 7 to be rate l,thereby selecting the motion 

this manner, the resolution in the zoom-up state is increased, priority mode which has low image quality but a large 

so that a point of interest can be observed in detail. number of frames. Then, processing halts until the next 

The image processing apparatus, which has the same idea 200111 operation is performed, 

as one of the foregoing arrangement (shown in FIG. 3), On the other hand, if NO in step S4, the flow advances to 
accormngtomeeiiibodimentofto 30 step S7 to check if the readout data is smaller than a 

described in detail below with reference to FIGS. 4 to 6. predetermined value p (mm). If YES in step S7, the system 

Herein, the same blocks in FIG. 4 and FIG. 5 as the ones controller 15 supplies a table write command to the quan- 

in FIG. 1 and FIG. 2 in terms of their functions are assigned tization coefficient table memory 1016 to read out a second 

the same reference numerals. quantization coefficient QF2 from the quantization coeffi- 
FIG. 4 is a schematic block diagram showing the arrange- 35 cient toWc ^"f" 3 ^ *016 and supplies the readout coefficient 

ment of the entire image processing apparatus, FIG. 5 is a to the quantization circuit 1005 in the same manner as 

block diagram showing the arrangement of a motion image described above. With this control, a quantization step rate 

encoding circuit 1208 in FIG. 4. and FIG. 6 is a flow chart f 0 ^ mat set based on the first quantization coefficient 

for explaining the operation of the image processing appa- Q F1 is sct 

ntus - 40 The flow then advances to step S9, and the system 

Referring to FIG. 4, execution of zooming of a video controller 1015 sets the data output rate of a buffer circuit 7 

camera 201 is instructed by a command externally input via t0 00 ratc 2, so that the frame rate assumes a value smaller 

a line 1253 or a command input from an operation unit (not than ^ Tanse 

shown )- On the other hand, if NO in step S7, the flow advances to 
The video camera 201 transfers image data to an image 45 stc P S10 to execute the same processing as in step S8. Thus, 

processing unit 202 according to the zooming command a third quantization coefficient QF3 is read out from the 

The transferred image data is subjected to image processing quantization coefficient table memory 1016 and is supplied 

and is stored in an internal buffer 1212 embedded in the to mc quantization circuit 1005. The flow then advances to 

image processing unit 202. Then the stored image data is stc P Sll to set frame rate 3. 

transferred to a motion image encoding circuit 1205. 50 With this processing, a fine quantization step is set, and 

Upon reception of the zoom execution instruction, a main hence, the image quality priority mode which has a low 

system controller 1207 supplies a zoom command to a frame rate but high image quality is set In this manner, the 

camera controller 203. The camera controller 203 keeps motion priority mode changes to the image quality priority 
supplying the zoom command to the camera 201 while it 55 mode as focal length changes from the wide end to the 

receives this command, thereby controlling the camera telephoto end. 

zoom operation. While the zoom command is kept supplied. In this embodiment, the compression mode is divided into 

the mam system controller 1207 reads out zoom driving three modes, but the present invention is not limited to this 

position 1 data from the camera 201 via the camera controller An H.261 image compression method has been exemplified! 

203, and stores the readout data in its internal memory 1211. ^ but an MPEG or other compression method can beadoptoL 

The operation wfll be described below with reference to In this ernbodiment, the operation upon input of a motion 

the flow chart shown in FIG. 6. image has been exemplified. Also, the presentin vention may 

A motion image encoding circuit 1208 communicates be applied to a case wherein a still image is compressed 

with the main system controller 1207 at predetermined according to a variable quantization step, as a matter of 

intervals to monitor if the zoom control is performed (step 65 course. 

2t liiJf^ ** 200m l con ( tro1 * startcd aDd * * room operation of the camera 201 is not limited to an 

also detected that the zoom control ends, a system controller optical zoom operation that changes the focal length by 
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^ a lens system but may be aliased by an electee 1 ^J^^ 

zoom operation that changes the readout range of an image ^^^^ wh^zooTfocal length to an 
from a CCD or a memory. Q £g£ fa smaUcr a predeteniiined wide-end focal 

Furthermore, image information input from the image le £T 
input unit 1000 is not limited to one output from the video 5 ^ ^ method according to claim 4, wherein the encod- 
camera, but may be externally input information. stcp includes a step of encoding image information at a 

As described above, according to the present invention, low compression ratio when the zoom focal length to an 
the encoding operation of the image information encoding object is larger man a predetermined telephoto-end focal 

unit is controlled in correspondence with the input operation length. . . . 

of image information by the image input unit, and the image 10 12. An image processing apparatus comprising. 

compression ratio is changed in correspondence with the input means for inputting image information 

image pickup state of an object by the image input unit. For optical zooming means for optically zooming the image 

this reason, when an object image is picked up in a zoom-up information input by said input means; 

state, the image compression ratio can be decreased to obtain ^ encoding means for encoding the input image inf arma- 

a high-resolution image, and the point of interest can be tion; and 

observed in detail. control means for controlling the encoding means on the 

As many apparently widely different embodiments of the basis of a zoom control amount of said optical zooming 

present invention can be made without departing from the means. . 

spirit and scope thereof, it is to be understood mat the ^ ^ apparatus according to claim 12, wherein the 
invention is not limited to the specific embodiments thereof zoom CO ntrol amount includes zoom-in and zoom-out con- 
except as defined in the appended claims. trol amounts. . 
What is claimed is: 14. The apparatus according to claim 13, wherein me 
1 An image processing method comprising: encoding means includes a means for encoding by decreas- 
an input step of inputting image information M ing a compression ratio of the input image inf ormation in a 
an optical zooming step of optically zooming the image zoom-in mode, and air^aiisfo^^ 

Con inpit b7 said input step; compression ratio of the input image information in a 

an encoding step of encoding the input image informa- according to ^ 13, wherein the 

tion;and optical zooming means increases a zoom focal length to an 

a control step of controlling me encc>dmg step 30 Sin a zcom-in mode, and decreases a zoom focal length 

of a zoom control amount determined at said optical zoon ^ >ut ^ 

zooming step. . . n A 16. The apparatus according to claim 13, wherein the 

2. The method according to claim 1, wherem me zoom 1* a ^ for inputting the image 
control amount includes zoonvin and zoom-out control ^ ^g^^ by dfywatring ^ numbcr Q f input frames per 
amounts. , it ti far ^ obiect in a zoom-in mode, and a means for 

3. The n^odaccordingtoclaiml wh^m me «cod^g u^J™ £ ^tSclation by increasing the number of 
step includes a step of encoding by decreasing a compres- 3J^^ t ^ fortte i je ctinaroorn-outmode. 
sionraUooftoeiimutimageurfai^^^ T 7 ^a^msTcording to claim 15, wherein me 
and a step of encoding by increasing ^^^ iatio „ control mcanVdbcreases the numbcr of input frames per unit 
of the input image informal., in a 40 a,,* m toe zoom-in mode, and increases the number of input 

4. THe method accordmg to ^J-^^"*° £££ per unit time in the worn-out mode. 

optical zooming step, a zoom focal lengto to an ^jert * ™™ apparatus according to daim 14. wherein the 
increased in a zoom-in mode and a zoom focal length to an J^f^j™ ^o,," quantization means size of 

obiect is decreased in a zoom-out mode. ^ . . . 

method according to daim * wherein the input 45 irny.*™*-^ aecoMng t0 ^ „, wherdn me 
step indudes a step of jS^^^^SZ * i^ means^udes a means for inputting image informa- 
decreasing the number of input ^P"^^ ^ inaeasing the number of input frames per unit time 

object in a zoom-in mode, and a ^^8*^ ft ezoomfocal length to the object is smaller than a 

information by increasing the numb« of input frames per j^j ^ 

unit time for the object in a zoom-out mode. m The aOTaratus according to daim IS, wherein the 

6. The method according to daim 4. wto^ at the ^ a IO£aas far in p Uttin g image toforma- 
control step, the number of input frames per unit MUme* ^'JT^easing the number of input frames per unit time 
decreased in the zoom-in mode, and the number of input ^^thVl^focal length to the object is larger than a 
frames per unit time ui-creasedin the ^^mode. ^^^3 focal lengto. 

7. The method according to claim 3. wherein the com- 5S ^^T^LjX^ccxjrdmg to daim 15, wherein the 
pression ratio is based on a quantization step size of image en ^ C ^^udes the means for encoding image 
information. . .... information at a hieh compression ratio when the zoom focal 

8. The method according to claim 4. wherantoe ^put g^oUS3«lh«« predetermined wide-end 
step indudes a step off inputting image information by ' Vi^rfh 

60 ^^apparatus according to daim 15, wherein the 
zoom focal lengm to the object is smaller than a predeter- ^J* ^ & fm encoding image 

mined wide-end focal lengm. information at a low compression ratio when the zoom focal 

9. The method according to claim 4. wh^b the mput jm is tager than a predetermined telephoto- 
step indudes a step of inputting image information by "f^.Tj^r 

decreasing the number of input frames per unit time when « enajocu ^ apparatus comprising: 

me zoom focal length to toe object is larger than a prede- f ormp3g bZ information; 

termincd telephoto-end focal length- mpm mcnua x^f- 6 
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zooming means for zooming the image information input zoom focal length to the object is smaller than a predeter- 

by said input means; and mined wide-end focal length. 

control means for controlling the frame number of input 50. Apparatus according to claim 25, wherein the input 

frames per unit time for an object on the basis of a zoom means includes means for inputting image information by 
control amount of said zooming means. 5 decreasing the number of input frames per unit time when 

24. Apparatus according to claim 23, wherein the zoom the zoom focal length to the object is larger than a prede- 
control amount includes zoom-in and zoom-out control termined telephoto-end focal length. 

an i? J ! in !f s * , . 31. Apparatus according to claim 25, wherein the encod- 

25. ^paran^ acccrduig to claim 23, further comprising ^g means includes means for encoding image information 
means for encoding the input image information. io * . . ^ L , ^ ~, 

26. Apparatus JLli3° K J^Eftte encod- t ^T^Zu^ Z °7 f °f * 
ing nxZs includesTmeLs for encoding byde^kga J^J 6 * h 8maUer ^ 8 P rcdttenmncd Wlde<nd focal 
compression ratio of the input image information in a TT* ^ , . — . 

zoom-in mode, and a means for encoding by increasing the . 31 A*^™*"^* to claim 25, wherein the encod- 

compression ratio of the input image information in a " Mg mcans includes racans for encoding image information 

zoom-out mode. at 8 low compression ratio when the zoom focal length to an 

27. Apparatus according to claim 25, wherein the input ^f** h man a Predetermined telephoto-end focal 
means includes a means for inputting the image information l^gth. 

by decreasing the number of input frames per unit time for 33. Apparatus according to claim 23, wherein the zoom- 

an object in a zoom-in mode, and means for inputting the 20 * n 8 means increases a zoom focal length to an object in a 

image information by decreasing the number of input frames zoom-in mode, and decreases a zoom focal length to an 

per unit time for the object in a zoom-out mode, object in a zoon>out mode. 

28. Apparatus according to claim 25, wherein the com- 34. Apparatus according to claim 33, wherein the control 
pression ratio is based on a quantization size of image means decreases the number of input frames per unit time in 
information. 23 the zoom-in mode, and increases the number of input frames 

29. Apparatus according to claim 25, wherein the input per unit time in the zoom-out mode, 
means includes a means for inputting image information by 

increasing the number of input frames per unit time when the * * * * * 
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